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Preface  to  the  First  Printing 

This  Information  System  for  Decision-Making  has  been  developed, 
pilot-tested  and  revised  over  a  period  of  some  eighteen  months.   The 
initial  development,  pilot-testing  and  final  editing  were  accomplished 
by  personnel  in  the  Research,  Planning,  Development  and  Evaluation  Component 
in  my  office.   The  University  of  Montana's  Division  of  Educational  Research 
and  Services  was  contracted  to  produce  an  audiovisual  orientation  package 
and  to  rewrite  the  original  manual.   My  special  thanks  go  to  the  school 
personnel  in  Simms,  Sun  River  and  Butte,  Montana,  who  graciously  cooperated 
in  the  pilot  tests. 


Cp&aM^-'a^  d^JJ^*-^^ 


DOLORES  COLBURG 
Superintendent  of  Public  Instruction 
Helena,  Montana 
March  1972 
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Preface  to  the  Second  Printing 

The  Information  System  for  Decision-Making  described  in  this  manual 
has  been  applied  in  several  different  situations  since  the  manual  was 
first  printed. 

People  in  ESEA  Title  III  projects  in  innovative  education  have  been 
using  the  system  to  evaluate  their  efforts.   The  Vocational  Education  Component 
in  the  Superintendent  of  Public  Instruction's  office  has  used  a  modified 
version  of  ISDM  to  serve  its  information  needs  and  simplify  reporting  procedures 
required  of  the  state's  schools.   The  authors  of  the  system  have  applied 
it  both  as  an  evaluation  and  a  planning  tool  in  providing  annual  services  to  the 
Migrant  Children  Education  Program  in  Montana.   Personnel  in  a  demonstration 
program  in  adult  education  have  used  the  model  for  organizing  information 
about  their  project  and  for  evaluating  it. 

This  manual  has  been  requested  by  educators  in  more  than  forty  states 
and  three  foreign  countries.   This  tremendous  response  and  the  variety  of 
applications  the  model  has  had  in  both  planning  and  evaluation  have  prompted 
us  to  go  to  a  second  printing.   We  encourage  educators  with  an  interest  in 
planning  and  evaluation  to  comment  on  the  system. 

ROBERT  A.  LEHMAN,  Ph.D. 
Acting  Director 

Research,  Planning,  Development 
and  Evaluation  Component 
July  1973 


1 


CONTENTS 

Pages 

INTRODUCTION  1-2 


PHASK  I  -  GETTING  STARTED 

1.1  Gaining  Commitment  3_^ 

1.2  Identifying  and  Examining  Policy  5-11 

1.3  Developing  an  Evaluation  Schedule   12-15 

1.4  Orienting  and  Training  Personnel  16-18 

PHASE  II  -  DESCRIBING  A  PROGRAM 

2.1  Identifying  Boundaries  19-24 

2.2  Describing  People  and  Program  Variables   25-33 

2.3  Describing  the  Decision  Structure   34-52 

2.4  Constructing  Objectives   53-63 

2.5  Constructing  Questions  64-70 

PHASE  III  =  GATHERING  THE  DATA 

3.1  Identifying  Data  Sources 71-77 

3.2  Selecting  Instruments  for  Evaluation  78-79 

3.3  Gathering  Conditions  80-83 

3.4  Organizing  Data 84-87 

PHASE  IV  -  PROVIDING  INFORMATION 

4.1  Analyzing  Data 88-94 

4.2  Reporting  Information   95-100 

INDEX  OF  SAMPLE  FORMS 101 


Digitized  by  the  Internet  Archive 

in  2011  with  funding  from 

IVIontana  State  Library 


http://www.archive.org/details/informationsyste1972mont 


INTRODUCTION 


Educators  everywhere  are  trying  to  keep  pace  with  the  rapid  social, 
economic  and  technological  changes  in  America  today.  A  fourth-grade  class 
studies  forest  ecology  at  an  instructional  center  located  in  uninhabited 
woods  twenty  miles  from  its  usual  classroom.   High  school  juniors  work 
half-days  for  a  month  in  community  businesses  and  receive  school  credit 
for  their  time.   Pre-school  children  from  disadvantaged  homes  are  provided 
with  experiences  that  will  help  them  compete  with  middle-class  children 
in  the  first  grade. 

These  examples,  and  hundreds  of  other  new  school  activities,  mark 
the  trail  modern  education  has  blazed  beyond  the  "Three  R's."  They  are 
ambitious  attempts  to  provide  our  younger  citizens  with  the  tools  and 
knowledge  needed  to  understand  and  live  in  the  ever-more-complicated, 
ever-more-challenging  world  around  us. 

The  new  and  exciting  ideas  in  modern  education  represent  what  is 
perhaps  the  greatest  strength  of  American  educators — a  willingness  to 
think  and  act  in  terms  of  the  future  of  the  children  they  serve.   Because 
of  this  orientation  to  the  future,  courageous  educational  policy-makers 
and  administrators  find  themselves  entering  uncharted  fields,  using  untested 
methods  and  materials.   Their  willingness  to  try  suggestions  made  by  parents, 
government,  industry  and  students  promises  continued  effort  to  achieve  truly 
useful  education  for  American  children. 

But  promises  are  not  enough.   The  challenges  of  the  times  invite 
skepticism.   This  skepticism  can  be  healthy  for  educators  if  it  is  not 
blindly  focused  on  past  educational  practices.   The  educators'  recognition 
that  perhaps  education  did  not  adapt  rapidly  enough  to  changes  in  the  past 
should  cause  them  to  examine  whether  efforts  at  improvement  are  suitable 
and  effective.   It  is  no  longer  sufficient  to  merely  assume  that  a  program 
is  meeting  the  needs  it  purports  to  serve. 

Administrators,  teachers,  parents,  community  members  and  students  all 
want  to  know  whether  a  program  is  succeeding  or  not  and  why  in  either  case. 
Among  the  many  programs  being  developed,  some  will  be  truly  effective.   The 
remainder  will  include  programs  no  better  than,  or  worse  than,  the  practices 
they  seek  to  improve.   That  crucial  distinction  will  not  be  made  unless  some 
means  of  determining  success  or  failure  becomes  an  integral  part  of  all  pro- 
grams. 

Determining  program  success  or  failure  is  not  a  new  idea  in  education. 
As  a  process,  it  has  been  called  various  names  which  cover  a  variety  of 
concepts.   It  has  evolved  from  "answering  to  your  boss"  into  a  much  more 
meaningful  concept  of  "evaluation." 
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The  Information  System  for  Decision-Making  (ISDM)  outlined  in  this 
manual  is  based  on  a  new  concept  of  evaluation.   The  word  "evaluation,"  as 
used  in  this  manual,  implies  a  process  of  identifying,  describing,  obtaining 
and  providing  useful  information  for  decision-makers,  so  that  they  may  better 
judge  and  choose  among  decision  alternatives.   In  the  past,  the  role  of  eval- 
uation has  been  to  approve  or  disapprove  a  completed  program.   This  approach 
robbed  the  process  of  its  greatest  potential--program  improvement.   However, 
as  a  system  of  providing  useful  information,  evaluation  can  help  the  decision- 
maker to  informed  judgments  on  needed  program  adjustments,  thus  assuring 
greater  possibility  of  program  success. 

The  assumption  which  underlies  this  new  concept  of  evaluation  is  that 
decisions  resulting  from  an  investigation  of  all  information  about  available 
alternatives  should  be  consistently  better  than  decisions  arrived  at  by 
guess  or  chance.   Decision-makers  who  agree  with  this  assumption  may  find 
the  evaluation  process  in  the  herein  described  Information  System  for 
Decision-Making  a  valuable  tool  for  improving  education. 

Who  are  the  educational  decision-makers  that  this  system  (ISDM)  might 
benefit?   Primarily,  they  are  those  individuals  who  recognize  that  deci- 
sions affecting  the  form  and  substance  of  instruction  are  made  on  many 
levels--student ,  teacher,  administrator,  parent,  local  board  member,  state 
and  federal  of f icials--and  those  who  believe  that  this  evaluation  system 
could  apply  to  a  broad  range  of  situations.   Effective  use  of  the  system 
involves  two  things:   the  decision-maker's  dedication  to  improving  program 
decision-making  and  his  personal  commitment  to  testing  ISDM  as  a  means  to 
that  end. 

The  ISIM  concept  of  evaluation  recognizes  an  unwelcome  fact.   Few 
decision-makers  have  either  the  time  or  the  training  to  gather,  analyze  and 
prepare  the  information  they  know  they  could  use  in  locating  and  judging 
alternative  courses  of  action.   In  practice,  this  task  is  usually  delegated. 
The  purpose  of  ISDM  is  to  systematically  organize  information-gathering 
efforts  in  an  educational  organization  so  that  program  status,  progress  and 
performance  can  be  examined  comprehensively.   The  task  involves  a  cooperative 
effort  between  decision-makers  and  evaluators  (information  providers).   Clear 
and  constant  communication  between  decision-makers  and  evaluators  is  a  necessity 
if  the  information  to  be  provided  is  to  be  accurate  and  pertinent. 

The  Information  System  for  Decision-MSking  has  four  phases.   In  the 
first,  a  commitment  to  evaluation  must  be  made,  various  policies  affecting 
the  operation  identified,  tasks  and  deadlines  agreed  upon,  and  personnel 
oriented  and  trained  in  the  system.   In  the  second  phase,  the  program  under 
evaluation  must  be  described  as  accurately  as  possible  in  order  to  have  a 
standard  against  which  to  compare  later  achievements,  changes,  improvements 
or  differences;  included  in  such  a  description  would  be  boundaries,  people 
and  program  characteristics,  objectives,  questions  about  those  objectives 
and  the  program's  decision  structure.   In  phase  three,  data  about  the  program 
is  gathered;  data  sources  are  identified,  instruments  for  data  gathering  are 
chosen  or  constructed,  a  useful  scheme  of  data  organization  is  developed.  And 
in  phase  four,  the  raw  data  is  transformed  into  useful  information  and  is  repor- 
ted to  the  decision-maker.   Depending  on  the  decision-maker's  needs,  the  system 
can  provide  him  a  single  report,  several  reports  or  even  a  continuous  flow  of 
pertinent  information. 
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PHASE  I  -  GETTING  STARTED 
1.1  Gaining  Commitment 

This  Information  System  for  Decision-Making  can  be  a  reservoir. 
It  can  generate  information.   It  can  store  information.   You,  the 
decision-maker,  can  gain  a  permanent  source  for  answers  to  your 
decision  questions. 

How  can  you  utilize  ISDM's  potential? 

ACTIVITY:  Once  convinced  of  the  information  system's  value, 
the  decision-maker  will  demonstrate  his  commitment  to  the  evalua- 
tion by  providing  funds,  time  and  personnel. 

CONSIDERATIONS:   Commitment  implies  the  following: 

(1)  understanding  the  concept  of  evaluation  as  it  is 
presented  in  this  manual; 

(2)  accepting  the  contention  that  decisions  based  on  valid 
information  will  be  sound;  and 

(3)  entering  an  agreement,  formal  or  informal,  to  begin  an 
evaluation. 
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1.1  Gaining  Commitment  (Continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


1.  Meets  with  you,  the 
decision-maker 


By  responding  to 
your  request  for  a 
conference  with 
someone  acquainted 
with  the  informa- 
tion system 


Explains  scope 
and  features  of 

ISDM 


By  using  orienta- 
tion materials  or 
through  conversa- 
tion with  you 


Gains  your 
commitment  to 
evaluation 


By  forming  an 
agreement  for 
evaluation  to 
begin 


1.2   IdentifyiriR  and  Examining  Evaluation  Policy 


If  ISDM  is  potentially  a  reservoir,  your  evaluation  policies  define  its 


dimensions . 


ACTIVITY:  An  evaluator  will  identify  and  examine  policy  defining 


his  role  and  responsibility  within  the  organization. 


CONSIDERATIONS:   Specifically,  the  evaluator  needs  the  following 


policy  information  (see  the  chart  on  Guidelines  for  Classification  of  Policy 


Statements  on  page  9) 


Access  to  data  sources 


(1)  Who  is  entitled  to  examine  records  and 
other  data  sources? 

(2)  Through  what  channels  does  the  evaluator 
gain  entitlement  to  data  sources? 


Access  to  data  base 
and  evaluation 
information 


(1)  What  information  is  available  to  the 
evaluator? 

(2)  How  is  availability  determined? 

(3)  How  will  the  information  be  used? 

(4)  What  are  the  criteria  for  determining 
the  use  of  information,  e.  g. , 
aggregation  (the  way  information  is 
combined) ? 

(5)  To  what  extent  must  names  and  confiden- 
tial materials  be  protected? 


1.2  Identifying  and  Examining  Evaluation  Policy  (continued) 


The  evaluator's  role 


(1)  What  authority  does  the  evaluation  team 
have  within  your  (the  decision-making) 
organization? 

(2)  What  authority  does  management,  the 
decision-maker  and  his  staff,  have  in 
conducting  the  evaluation? 

(3)  Under  what  conditions  may  consultants 
or  contracted  service  agencies  be 
employed? 

(4)  How  would  evaluation  staff  positions 
be  placed  on  an  organizational  chart? 


Resource  limitations 


(1)  What  are  the  responsibilities  of  the 
evaluation  team  and  management? 

(2)  What  are  their  staffs'  responsibilities? 

(3)  What  are  the  precise  duties  (job 
descriptions)  of  evaluation  staff? 


Scheduling 


(1)  What  are  procedures  for  handling 
evaluation  requests? 

(2)  How  are  time  schedules  of  the  eval- 
uation team  and  data  sources  to  be 
coordinated? 


Reporting 


(1)  What  standards  are  followed  in  prepar- 
ing reports? 

(2)  What  are  the  criteria  for  determining 
what  information  is  included  in  reports? 

(3)  How  are  reports  to  be  disseminated  to 
your  organization? 

(4)  What  are  the  procedures  for  releasing  and 
publishing  evaluation  material  and  reports? 


1.2   Identifying  and  Examining  Evaluation  Policy  (continued) 

It  is  within  these  seven  policy  areas  that  authority  for  existence 
of  an  evaluation  team  and  conduct  of  an  evaluation  resides. 

If  these  policy  areas  have  been  defined  in  writing,  the 
evaluator  need  only  collect  and  examine  written  statements  in  order 
to  identify  authority.  When  policy  statements  do  net  exist  in  these 
areas  or  conflict  with  each  otherj  the  evaluator  must  ask  your  assis- 
tance in  (1)  generating  policies  necessary  for  evaluation  or  (2) 
resolving  conflicts  among  existing  policies.   In  some  cases ^  the 
evaluator  may  have  to  request  special  authorization  to  conduct  activities 
undefined  In  policy  statements.   For  example,  if  your  organization  -.as  no 
policy  statement  on  access  to  confidential  data  from  student  academic 
records  and  the  evaluator  needs  data  of  this  type,  he  must  di=:termine  who 
has  authority  to  release  the  information  and  must  request  its  release. 


1.2   Identifying  and  Examining  Policy  (continued) 

WORK  BREAKDOWN  FCR  DETERmNING  EXISTING  POLICY  ON  EVALUATION 
Who  Does  What  How 


Evaluator 


Collects  available  statements 
of  policy  governing  the 
organization  and  the  program 
under  study 


Completes  and  files  forms 
classifying  policy 
statements 


By  making  prior 
arrangements  with 
appropriate  organ- 
izational and 
program  personnel 

By  using  Forms  1 
and  2  (samples  on 
following  pages) 


Who 


PROCEDURE  WHEN  EVALUATION  POLICY  DOES  NOT  EXIST  OR  CONFLICT 
Does  VHiat  How 


Evaluator 


Determines  if  policy 
statement  exists  for  conduct 
of  assigned  activity 


2.  If  policy  exists,  reports 
it  as  the  authority  to 
proceed 

3.  Proceeds  with  activity 

4.  Reports  lacking  or 
conflicting  statements 
of  policy  to  appropriate 
evaluation  and  program 
personnel 

5.  Jeteraines  with  policy- 
maker the  authority  to 
proceed 

6.  Assigns  or  reassigns 
activity 

7.  Classifies,  codes  and  files 
statement  of  resulution 


By  referring  to  the 
form  used  to  record 
applicable  policy 
statements  (Form  1, 
page  10) 

By  writing  memo 


According  to  plan 

By  writing  memo  or 
scheduling  meetings 


By  submitting  Form  2 


By  conferring  with 
staff  member 

According  to  estab- 
lished procedure 
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(This  form  completed 
as  an  example.) 


FORM  1 


CLASSIFICATION  OF  POLICY  STATEMENT 

INSTRUCTIONS:   Form  1  will  be  used  for  classification  and  storage  of  policy 
statements.   Statements  are  to  be  taken  from  policy  documents  obtained  in 
Steps  1-2  of  work  breakdown  for  Section  1.2.   Statements  are  selected  and 
classified  by  topic.   "Topic"  refers  to  the  topics  mentioned  in  the  narrative 
to  1.2.   The  topic  code  and  sub-topic  code  are  obtained  by  referring  to  the 
Guidelines  Chart.   "Source"  refers  to  the  policy  document  from  which  the 
statement  was  taken.   Each  statement  is  to  be  written  verbatim  on  this  form. 


Topic; 


Rurlget.    nnrl    R^ftnuy^^o    T.n'.mi\-h.n+.i'nvis 


Date: 


10/21/72 


Topic  Code:_ 


4 


Sources :  School  District  #1  Governing  Codes 


Statement 


Sub-Topic  Code 


"The  Board  shall  approve  any  contract  made  between 
any  person  or  organizations  not  employed  by  School 
District  #1  and  any  school,   commission,   or  person 
that  is  part  of,   or  employed  by.   School  District  #1." 
(Governing  Code  5.12A) 


4,3 
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STATEMENT  OF  AUTHORITY  TO  PROCEED 
(Form  completed  as  an 
example.)  FORM  2 

INSTRUCTIONS:   Form  2,  Part  A,  will  be  submitted  by  the  evaluator  to  the  program 
policy-maker  to  obtain  authority  to  proceed  with  evaluation  in  a  situation  in 
which  there  is  either  a  lack  of  policy  or  in  which  written  policy  statements  con- 
flict.  Part  B  will  be  completed  by  the  program  policy-maker  and  returned  to  the 
evaluator.   A  separate  form  should  be  prepared  for  each  situation. 

PART  A  (To  be  completed  by  evaluator) 

To:    John  B.    Chairman 


In  our  review  of  policy  statements  of  your  program  we  find  that 

I    I  No  statement  of  policy  exists 

I  X    I  Conflicting  statements  of  authority  exist 

regarding  the  following  topic,  as  described: 

Sahocl  Distriot  #1  Governing  Code  Number  3.41B  states  that  "the  Board 

shall  determine  which  persons  may  revieu)  personnel  files.  "    At 

Waskvngtcn  Junior  High^   the  principal  reserves  the  right  to  grant 

persmission  to  inspect  personnel  files. 

For  the  evaluation  to  proceed,  authority  should  be  granted  to 


Robert   I.  Ei^aluator in  order  that  he  may  inspect  the 

Washington  Junior  High  personnel  files  and  complete  a  description  of 

information  available  on  math  teachers  in  School  District  ifl„ 

Please  supply  a  statement  of  policy  that  will  (clarify/define)  a  resolution  of 
this  conflict. 

Date      3/21/72    Name  Robert  L.   Evaluator        Title  Evaluation  Coordinator 


PART  B  (To  be  completed  by  program  policy-maker(s) .   Statement  of  authority  to 
proceed:   (If  resolving  a  conflict  of  existing  policies,  please  describe  speci- 
fically how  those  policies  are  to  be  amended.) 

Evaluators  participating  in  the  study  of  School  District  #1  math  programs 

shall  have  acaess   to  the  personnel  files  in  all  schools .      The  prinaipal 

of  Wnpshtlngtnn  Junior  High  Snhnnl  hjiR   haan  sn  infnrmp.d. 

Date     .^/ag/72  Name  John  B.    Chairman Title  Board  Chairman 


II 


1.3   Developing  an  Evaluation  Schedule 

A  master  schedule  for  evaluation  functions  much  as  a  building  contract. 

In  building  the  ISDM  reservoir,  the  master  schedule  lists  what  work  is  to  be 

completed  by  what  dates.   In  this  case,  it  also  lists  the  personnel,  who 

like  sub-contractors,  will  be  assigned  specific  responsibilities  in  the 

overall  project. 

ACTIVITY:   An  evaluator  will  develop  a  master  schedule  (time  line) 

for  completing  the  activities  defined  in  this  ISDM  manual  and  will  assign 

tasks  to  personnel. 

CONSIDERATIONS:   Cooperation  between  management  and  evaluation  staffs 

is  extremely  important  since  the  master  schedule  involves  identifying  and 

assigning  tasks.   The  evaluator  will  provide  answers  to  the  following 


questions: 


Activity 


Which  tasks  will  be  necessary  to 
implement  ISDM,  and  in  which  order 
should  they  be  accomplished? 


Assignment 


Which  personnel  are  best  suited  to 
accomplish  these  tasks? 
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1.3  Developing  an  Evaluation  Schedule  (continued) 


Time 


Who  will  be  available  and  how 
much  time  can  they  spend  on 
evaluation  tasks? 


Cooperation  in  Scheduling 


How  can  evaluator  and  staff 
schedules  best  be  coordinated? 


Further,  the  master  schedule  will 


(1)   set  major  deadlines. 


(2)   show  relationships  among  activities  to  be  completed  in 


ISDM,  and 


(3)   provide  a  concise  breakdown  of  all  steps  in  the  system. 


An  example  of  a  master  schedule  follows, 
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1.3  Developing  an  Evaluation  Schedule  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


Develops  a  master  schedule 
for  completing  evaluation 
tasks 


By  identifying  tasks 
to  be  performed, 
including  those 
throughout  the  ISIM 
manual;  identifying 
resources  and  personnel 
available;  and  assign- 
ing tasks  and  completion 
dates 
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1.4  Orienting  and  Training  Personnel 


Informed,  trained  personnel  will  build  the  ISDM  reservoir  which  will 


hold  the  resource-- informal ion--vital  to  evaluation. 


ACTIVITY:  An  evaluator  will  orient  personnel  to  the  application  of 


evaluation  to  decision-making  and  will  train  personnel  in  the  use  of  ISOM. 


CONSIDERATIONS:   Evaluation  orientation  and  training  must  communicate 


the  following  ideas: 


(1)  Evaluation  is  a  process  for 

identifying,  describing,  obtaining 
and  providing  useful  information  to 
decision-makers  for  use  in  judging 
decision  alternatives. 


(2)   Providing  answers  to  questions 

that  the  decision-maker  considers 
Important  is  the  major  function 
of  ISDM. 


(3)   ISDM  employs  a  workable  method 

for  producing  Information  leading 
to  identification  of  needs  and 
barriers  to  success. 


Audience 

Any  person 
associated 
with  the 
evaluation 
project 
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1.4  Orienting  and  Training  Personnel  (continued) 

After  orientation,  the  evaluator  must  train  personnel  who  will  have 
specific  roles  in  the  implementation  of  the  information  system.   Personnel 
must  be  shown  what  the  information  system  can  do  and  what  is  required  of 
individuals  operating  the  system. 

ISDM  requires  specific  understanding  by  all  personnel  who  will  be 
working  with  it.  Informal,  open  orientation  and  training  sessions  and 
question-and-answer  sessions  combined  with  visual  aids  are  recommended.* 


*A  slide-tape  Training  Packet  for  ISDM  is  available  from  the  Office  of 
the  Superintendent  of  Public  Instruction,  Helena,  Montana  59601. 
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1.4  Orienting  and  Training  Personnel  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


Acquaints  decision-maker 
with  importance  of  evaluation 
as  a  decision-making  aid 


By  giving  informal 
presentations  and 
interviews 


Trains  those  persons  who  will 
use  the  information  system 


By  giving  more  formal 
presentations,  usually 
designed  to  include  use  of 
training  packages 


Orients  all  who  will  perform 
tasks  using  the  Information 
System  for  Decision-Making 


By  an  open-ended  program 
that  would  combine  question 
and  answer  sessions  and 
visual  aids  presentations 
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PHASE  II  -  DESCRIBING  A  PROGRAM 
2.1   Identifying  Boundaries 

The  land  topography,  water  source  and  dam  all  place  boundaries  on  a 
reservoir,  just  as  various  legal,  operational  and  financial  structures  place 
boundaries  on  your  program. 

ACTIVITY:   The  evaluator  will  obtain  a  general  description  of  the 
program  under  study,  based  on  whatever  factors  fix  or  limit  that  program. 

CONSIDERATIONS:   Boundaries  are  legal,  operational  or  financial 

Limitations  which  affect  activities  within  a  program.   All  boundaries  must 

be  Identified  before  the  evaluator  can  describe  the  actual  program.   Possible 

limiting  factors  include: 

Legal  Boundaries:       federal,  state,  or  district  rulings, 

regulations  or  policies; 

Operational  Boundaries:  program  objectives  or  activities;  and 

Financial  Boundaries:    program  costs,  federal,  state  or  district 

funds  available. 


19 


2.1   Identifying  Boundaries  (continued) 

Each  boundary  description  of  a  program  should  answer  the  following 

questions: 

What  is  the  name  of  the  program? 

What  is  the  program  authority? 

What  can  the  program  do?  What  can't  it  do? 

What  persons  or  offices  will  benefit  from  program  activities? 

What  method  or  procedure  does  the  program  follow? 

Who  applies  the  method? 

What  is  the  geographical,  educational,  demographic  scope  of  the  program? 

How  much  does  the  program  cost  and  what  is  the  source  of  funding? 

What  schedules  and  deadlines  must  be  met  by  the  program  personnel? 

What  other  programs,  if  any,  come  under  this  program's  supervision? 

Who  can  supply  further  information? 
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2.1   Identifying  Boundaries  (continued) 

The  preceding  examples  are  likely  to  apply  in  all  evaluation  cases. 
However,  additional  program  boundaries  may  be  identified  as  program 
questions  are  formulated  in  Section  2.5. 

To  facilitate  the  collection  of  boundary  data,  special  forms  should 
be  filled  out  by  appropriate  program  personnel  and  made  available  to  the 
evaluator  (see  form  4  on  pages  23  and  24) . 
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2.1   Identifying  Boundaries  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


Explains  to  appropriate 
program  personnel  the  kinds 
of  data  that  are  required  for 
the  purpose  of  identifying 
boundaries 


By  scheduling 
appointments 


Program 
Personnel 


Fill  out  form  4 


By  providing  the 
boundary  data 
requested  by  the 
evaluator 


Evaluator 


Meets  with  program 
personnel  to  collect 
forms,  answer  questions 
and  hear  comments 


By  scheduling 
appointments 


Examines  completed  forms  to 
determine  whether  necessary 
data  has  been  provided. 
Recycles  to  Step  1  if  data 
is  insufficient 


By  comparing 
available  data 
with  boundary 
description 


Prepares  data  for  appropriate 
file 


By  editing,  rewriting 
as  necessary  to 
assure  uniformity 
of  style 


Meets  with  appropriate 
program  personnel  to  secure 
approval  or  non-approval  of 
edited  material.   Recycle 
to  preceding  step  if  necess0ry 


By  scheduling 
appointments 


Obtains  final  approval  and 
files  boundary  data 


By  obtaining 
signature  of  the 
dec is  ion- maker 
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(Form  completed  as  an 

illustration.)  FORM  4 


SUMMARY  OF  A  PROGRAM 

1.  Title  and  Number  of  Program: "AIM"  Achievement  Improvement  in 

Mathematics ______^__^_______^____^__^_^_________________ 

2.  District (s)  Participating:   Include  name,  number  and  county. 

District  #1^   Lyndale^  Montana   -  Alkali  County 

3.  Intent/Purpose:  What  is  the  prograin(s)  to  provide,  assist,  regulate 
etc.?  For  what  reason,  with  what  goals  and/or  governed  by  what  philosophy? 

To  improve  attitudes  and  achievement  in  mathematics  for  under- 

achievers  and  slew  and  reluctant  leaxners 


4.   Audience  Identification:   Specify  persons  or  offices  to  benefit  from  your 

activities  (e.g.,  teachers,  handicapped  students,  districts,  school  staff,  etc.) 

Designed  for  9th  grade  alau  and  reluctant  students  in  mathematics 


5.   Execution/Procedure:  How  is  each  project  carried  out?  Through  what  agencies 
and/or  organizations?   Involving  what  staff  members  (by  title)? 

Students  are  involved  in  a  mathematics  laboratory  approach  to  learning 

tc  give  them  "handson"  experience  and  to  allow  them  tc  yarticiyate  in 

a  mental,  physical  and  social  learning  process.     The  program  is  using 

matsrials  develoved  hit  similar  programs  in  other  states  and  oommerical 

■pre ducts  as  well  as  materials  developed  locallu 


Henj-v  Fleming:     Federal  Education  Director 


Alex  Greaom:     Procrram  Supervisor 


Pilot  Teachers:     James  Allen 


Shirley  Erite 


June  Moritz 


Richard  Roe 
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SUMMARY  OF  PROGRAM  (PAGE  TWO) 

6.   Scope:   In  what  areas  does  the  project(s)  operate  (geographical,  educational 
demographic) ? 

Curvently  serving  approximatelij   280  9th  grade  Lyndale  Public  School 

students..      Next  year  to  be  expanded  to   lOth  grade  and  later 

disseminated  statewide, 


7.   Finances:   Indicate  source,  amount  and  duration  of  funding. 

Duration  of  funding  is  Julu   I.    1970,  to  June   SO,    l97Uunder  ESEA  Title  III 

in  the  amount  of  $52.645,00.      Program  may  make  application  for  a 

cnntinuation  grant  for  a  maximum  of  two  additional  uears. 


8.   Deadlines/Schedules:   What  dates  and/or  time  limitations  are  important  to,  or 
must  be  observed  by,  the  program  and  its  participants? 

ESEA  application  deadlines:     April   l^    1971;   April   I,    1972 

Summer  workshop:     June   21,    1971  -  August   Z.3,  1971 

Materials  ready   to  begin  dissemination:     September   I,    1971 

Unlv';rstty  of  Eureka  Evaluation  Completion:     March  26,    1971 

Fvnal  Program  Report:     June  20,    197 Z 


9.   Contact:   Name  and  position  of  staff  member  who  may  be  contacted  for  additional 
information  or  clarification  about  the  program. 

Alex  Gregory ,   Progam  Supervisor 

1224  Warren.   Lyndale,   Montana  59666 


Date:    2/9/71         Completed  by: Evaluation  Coordinators 
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2.2   Describing  People  and  Program  Variables 

Two  major  considerations  in  building  a  reservoir  are  the  water  supply,  and 
how  it  flows.   Similarly,  the  supply  and  flow  of  valuable  information  vithin 
your  organization  affect  the  building  of  an  ISDM  evaluation. 

ACTIVITY   The  evaluator  will  describe  the  program  in  terms  of  its 
organization,  activity  and  content,  methods,  facilities  and  cost,  and 
personnel  according  to  demographic  characteristics. 

CONSIDERATIONS:  This  description,  recorded  on  ISDM  forms,  will  chart 
both  available  prograkii  information  and  the  flow  of  that  information  within 
your  organization.   (.See  forms  5,  6  and  7.) 

For  his  description,  the  evaluator  must  determine  and  then  record  who 
has  information  and  wnere  it  is  located  so  that  it  may  be  easily  collected 
later.   Idenc if icat lot  of  policy  (Section  1.2)  and  boundaries  (Section  2.1) 
will  be  useful. 
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1.1      Describing  People  and  Program  Variables  (continued) 


People  Variables 


Tlie  evaluator  will  record  data  about  persons  or  entities  affecting 


your  program;   students,  teachers,  administrators,  specialists,  family  and 


community,  for  example.   At  this  point,  the  evaluator  will  determine  only 


what  kinds  of  demographic  data  are  available;  specific  demographic  data 


may  be  required  later  in  the  evaluation. 


To  illustrate,  the  evaluator  might  need  to  know  if  the  following  types 


of  demographic  data  are  available  for  each  of  the  persons  or  entities 


described; 


Administrators,  Teachers,  Specialists 


Students 


educational  background 

work  experience 

salary 

prof fe.-^sional  affiliations 

riLin-prof essional  affiliations 

Lther  data 


age 

grade  level 

sex 

achievement 

health  (physical,  mental, 

emotional) 

other  data 


Fami ly 


Community 


size 

agr; 

distribution 

income 

residence 

education 

mobi 1 i ty 

ether  data 


geographic  setting 

history 

population 

economic  and  social  structures 

government 

power  structure 

socioeconomic  stratification 

commercial  and  financial  status 

other  data 
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2.2   Describing  People  and  Program  Variables  (continued) 

A  special  form  indicating  the  name  of  the  person  or  entity  being 
described  and  listing  what  data  is  available  from  what  sources  is  recommended. 
(See  sample  form  5.) 

Since  information  on  this  form  will  be  utilized  in  the  decisions, 
objectives  and  questions  (Sections  2,3,  2.4,  2.5)  of  your  program  evaluation, 
the  ev/aluator  will  record  it  as  accurately  and  thoroughly  as  possible.   The 
exact  title  for  each  person  or  entity  will  be  recorded;  data  source  will  be 
described  fully,  so,  if  necessary,  the  evaluator  can  return  to  that  source; 
data  will  be  precisely  labeled.   For  example,  if  an  aptitude  test  score  is 
parr  of  the  data  on  a  pupil,  the  name  and  form  of  that  test  should  be  recorded. 

Program  Variables 

A.  similar  me.;hod  will  be  used  to  describe  program  variables: 
orgartiz  dtion,  activity  and  content,  method,  facilities  and  cost. 
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2.2   Describing  People  and  Program  Variables  (continued) 


It  should  be  re-emphasized  that  at  this  point  the  evaluator  is 
i.H)nicrncd  with  describing  rather  than  collecting  data  on  factors  affecting 
your  program.   Below  are  examples  of  the  kinds  of  data  he  needs: 


Organization:   organizational  chart,  description  of  work  division  within 

program; 


Activities  and  Content:   activities--description  of  planning  activity, 

(e.g.,  review  of  current  programs,  definition  of  needs  and  opportunities, 
identification  of  programs,  study  of  proposed  solutions,  etc.), 
description  of  tasks  performed  to  implement  the  program  (e.g.,  how 
planning  objectives  and  program  coordination  are  achieved) ;   content-- 
topical  description  of  subject  matter  (e.g.,  a  listing  of  concepts 
emphasized  in  the  program); 


Method :   description  of  approaches  (e.g.,  practice-oriented,  machine- 
aided,  media-supplemented,  teacher-guided,  student-guided,  etc.); 


Facilities :   description  of  space  and  equipment  (e.g.,  regular  classroom  with 
Free  Response  Teaching  Machines,  8  mm  single  concept  loop  films, 
calculators,  etc.);  and 


Cost :   description  of  itemized  and  total  budget  on  facilities,  maintenance 
and  personnel. 
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2.2   Describing  People  and  Program  Variables  (continued) 


WORK  BREAKDOWN 


Who Does  What How 

Evaluator      1.   Determines  and  records         By  reviewing  Sections 

who  has  information  on         1.2  and  2.1 
people  or  entities  affecting 
the  evaluation  program 
and  records  the  location 
of  information 


2.   Determines  and  records         By  using  form  6 
what  data  is  available 
on  the  program  under  study, 
organization,  activity 
and  content,  method, 
facilities  and  cost, 
and  records  the  location 
of  data 


Determines  and  records  what    gy  using  form  5 
data  is  available  on  the 
people  in  the  program 
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DESCRIPTION   OF    PROGRAM 
FORM   6 


i 


INSTRUCTIONS:   This  forni  is  oeslgned  to  provide  the  evaluator  and,  subsequently,  the 
docifsion-makcr  with  a  clear  description  of  a  project.   Consequently, 
the  person  completing  the  form  should  respond  to  each  item  in  sufficient 
detail  to  fully  describe  the  program.   It  is  suggested  that  the  respon- 
dent ask  for  a  review  of  the  completed  form  by  a  staff  member  outside 
his  program.   This  review  should  serve  as  a  clarity  check. 

1.   Title  and  number  of  Program 

"A  TM"     AnhJp.vement  Improvement  in  Mathematics 


1 


Organization 

Describe  your  program  by  constructing  an  organizational  chart  Indicating  the 
horizontal  and  vertical  structures  (i.e.,  parts  of  an  organization  at  the  same 
level  and  the  up  and  down  dimensions  of  an  organization) . 


Superintendent 


Assistant 
Superintendent 


Principals 


Program 
Supervisor 


Federal 
Program  Dir 


Teachers 


Program 
Supervisor 


Students 


Assistant 
Superintendent 


2    Principals 


5    Teachers 


Students 
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DESCRIPTION  OF  PROGRAM  (PAGE  TWO) 

Content 

Describe  the  structure  or  body  of  knowledge  which  is  identified  with  the 
subject  matter  of  the  discipline.   It  could  be  described  through  a  structure 
of  topics  to  be  covered  within  and  across  grade  levels  (e.g.,  compo&ed  of 
eleven  nathematical  concepts  with  emphasis  placed  on  vocabulary  and  applica- 
tion of  these  concepts). 

i/hicle  numbersj  fractions^    ieaimalSi  per  aent,  pre-algebra,  algebra 

n>^/l  ■janmotr-^  npo    nnno.fflA  fr-^m  a   r>P.al  hns^in.  npprnanh   to   tha  more 

QompLex  operation  needed  in  performing  the  abcve  ob.jeotives,         


Me*".hod 

Describe  the  process  or  processes  used  to  implement  the  content  described  in 
Item  j;  (e,g,,  individualized  instruction,  multi-media  approach,  non-graded 
class  structure,  supplementary  teacher  training,  etc.). 


The 

labsrat. 

ly  apprca.2 

h  is  used  with 

a 

"hands  on"  expe 

rlenae  fc"  the 

pupils  with 

an  interee 

t  being  placed 

en 

the 

student's 

own  needs. 

Mathemat'.ca. 

.  game  8 

ana 

'  wanipulction  c 

ieviaea 

are  used  t 

?  strengthen 

the 

baeio  at 

■■naepts 

^r 

the  pr:>gram. 

Facilities 


Describe  the  space,  special  equipment  and  expendables  pertaining  to  your 
program  (eg.,  defined  b>  a  regular  classroom  setting,  the  Free  Response 
Teaching  I'iachines,  prototype  electronic  machine  8  mm  single  concept  loop 
filxs  and  calc.-lot\3rs)  .. 

P.  nla^RY':   ma   r..v.d  ^.rjujpmp.nt ;      CaiJulatnrR  are   :^ed  to   ih^.ak  hasif-  skills.. 

Tablee  and  chairs  replaae  desks,     A  Thermafax  noyier  is  utilized  in  the 

pr:Qvani.     A  file  cabinet j,   a'  :nq  with  stcrage  shelves  are  also  necessary. 


DESCRIPTION  OF  PROGRAM   (PAGE  THREE) 

6.   Cost 

How  much  money  is  required  £or: 

a)  facilities  $30,000 

b)  maintenance  $  3,645 

c)  personnel  $20,000 

d)  TOTAL  $53,645 

Date    ff/7/77    Completed   by  Jnhy,    nno.^    T<^inlimtnr> 
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2.3  Describing  the  Decision  Structure 

Surveys  of  purpose,  site  and  water  flow  are  basic  to  the  work  of  those 
designing  reservoirs. 

Descriptions  of  purpose  (change  situation) ,  site  (program  organization) , 
and  flow  (information  flow)  are  used  by  evaluators  in  planning  an  information 
system.   In  combination,  these  descriptions  define  a  program  decision  structure. 

ACTIVITY:   The  evaluator  will  describe  the  program  decision  structure. 

CONSIDERATIONS:   Information  needed  to  describe  a  decision  structure 

includes 

boundaries  within  which  decision-making  occurs, 

decision-making  procedures  and  activities  within  these  boundaries, 

roles  and  functions  of  program  decision-makers, 

information  currently  used  by  program  decision-makers. 

In  the  ISDM,  three  surveys  provide  the  data  from  which  the  description  of 

program  decision-making  is  compiled.   The  nature  of  these  surveys  is  described 

in  the  following  chart. 


DESCRIPTION  OF 
PROGRAM  DECISION-MAKING 


Legal  authority 

and  responsibility 
of  the  program 


> 


A  statement  of  laws  and/or  policies  specifying 
duties  and  responsibilities. 
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2.3  Describing  the  Decision  Structure  (continued) 


The  change_ 
situation 


> 


A  classification  and  anal  sis  of  the  purposes 
and  objectives  of  programs  under  study,  e.g., 
ESEA  Title  I  would  be  classified  as  a  federal 
legislative  action  (influence  for  change) 
designed  to  change  the  national  school  system 
(unit  of  change)  by  supporting  local  district 
programs  designed  to  meet  the  needs  of 
"educationally  deprived  or  disadvantaged" 
students  (subjects  of  change) . 


Decision  roles  and_ 
functions 


> 


An  identification  of  the  decision-making 
hierarchy  (organizational  charts,  etc.). 
The  charting  of  decision  roles  as  related  to 
eleven  basic  decision  functions. 


Information 


> 


Locating  forms  and  reports,  determining 
storage  method  of  forms  and  reports, 
defining  flow  of  information,  defining 
audiences  for  information,  stating  when 
information  will  be  needed. 


Each  survey  is  discussed  in  some  detail  later  in  the  Section.  The  infor- 


mation in  these  surveys  is  used  to  answer  the  following  questions  fundamental 


to  information-system  planning: 


Vfho  has  legal  authority  for  decision-making? 


Who  has  delegated  responsibility  for  decision-making? 


What  are  the  decision  influencers  (influence  for  change)? 


What  are  the  decision  racifiers  (subjects  of  change) 


vlho   are  the  probable  audiences  for  information? 


With  what  level  of  information-'-detailed  report,  summary,  media 
presentation--are  audiences  to  be  provided? 
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2.3  Describing  the  Decision  Structure  (continued) 

When  is  the  information  needed? 

Can  the  evaluator  meet  the  decision-maker's  timetable? 
Is  it  technically  feasible  to  provide  the  information? 
What  information  is  already  available? 


What  is  the  change  setting  (large  change  with  small  amount  of 
supporting  information,  small  change  with  small  amount  of  supporting 
information,  small  change  with  much  supporting  information)? 


SETTING  BOUNDARIES  FOR  PROGRAM  DECISION-MAKING 
The  boundaries  of  decision-making  include  legal,  operational  and  financial 
limits.  These  limits  are  classified  and  reported  on  form  9,  Report  of 
Program  Change  Situation.   Sources  of  information  useful  in  completing  this 
summary  form  include 

statements  of  law  and/or  policy  governing  program  operation, 
form  4,  Summary  of  Program  Operation, 
form  6,  Program  Description, 
a  list  of  program  objectives. 
The  purpose  of  this  form  is  to  summarize  the  factors  affecting  decision- 
making in  the  program  organization. 

Four  classes  of  factors  are  considered:   purposes,  influences  for  change, 

units  of  change  and  subjects  of  change.   Purpose  statements  may  be  found  in 
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2.3   Describing  the  Decision  Structure  (continued) 


program  law,  policy  and  objectives.   Some  examples  of  influences  for  change,  units 


of  change  and  subjects  of  change  are  as  follows: 


INFLUENCES 

FOR   

CHANGE 

scientific  lirerature 

technology 

authorities 

experts 

legislative  action 

community 

professional  staff 


UNITS 

OF   

CHANGE 


> 


classroom 

school 

local  school  system 

county  school  system 

state  school  system 

national  school  system 


SUBJECTS 

OF 

CHANGE 

students 

teachers 

administrators 

support  staff 

curriculum 

methods 

materials 

schedules 

physical  resources 

objectives 

policies 

values 

time  allotment 

organization 


Examples  classified  as  subjects  of  change  may  operate  as  influences  for 
change  and  vice  versa.   To  illustrate:   To  improve  the  reading  performance  of 
students  (subject  of  change)  in  a  third-grade  classroom  (unit  of  change),  a 
principal  (influence  for  change)  may  teach  his  staff  (subject  of  change)  new 
methods  (subject  of  change) .   Or  students  (influence  for  change)  might  attempt 
to  influence  action  (subject  of  change)  of  the  state  legislature  (unit  of  change) 

DECISION-MAKING  IN  PROGRAM  OPERATION 

Having  set  the  decision-making  boundaries  for  a  program,  the  information 
system  designer  next  needs  to  find  how  the  work  of  decision-making  is  being 
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2.3  Describing  the  Decision  Structure  (continued) 

accomplished  for  the  program.   Sources  of  information  for  this  description  are 
the  organizational  chart  and  the  data  obtained  from  program  personnel  on  form  8, 
Description  of  Decision  Structure. 

The  data  from  form  8  is  used  to  classify  the  role  and  functions  of  each 
decision-maker.   The  roles  are: 

R"~responsibility  for  initiating  and  carrying  through  decision  action, 

I — informing  others  of  decisions, 

A — approving  decisions  made  by  someone  else,  and 

C--consulting  on  decisions. 
The  functions  are: 

1.  defining  a  population  to  be  served, 

2.  setting  program  objectives, 

3.  changing  program  objectives, 

4.  setting  relative  importance  of  program  objectives, 

5.  changing  relative  importance  of  program  objectives, 

6.  establishing  and  replacing  policy, 

7.  establishing,  modifying  or  replacing  a  program. 
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2.3  Describing  the  Decision  Structure  (continued) 


8.   employing,  assigning  and  replacing  personnel. 


9.   budgeting,  purchasing  and  accounting. 


10.   collecting  and  disseminating  information,  and 


11,   evaluating. 


A  sample  chart  illustrating  how  roles  and  functions  are  related  follows: 


DECISION  STRUCTURE  CHART 
Program:   Reading 

Organization 
Role 

Function 

Examples  (from 

decision  functions 

on 

preceding  page) 

Setting     Changing 
program     program 
objectives  objectives 

Setting  relative 
importance  of 
program  objectives 

Changing  relative 
importance  of 
program  objectives 

Board  of  Trustees    A 

lA 

lA 

lA 

Superintendent 

lA 

lA 

lA 

lA 

Principal 

lA 

lA 

lA 

lA 

Program  Director 

IR 

CAI 

IR 

CAI 

Teacher 

CA 

IR 

CA 

IR 

Specialist 

C 

C 

C 

C 

R---Initiates 

I-=Informs 

A- -Approves 

C — Consults 

Using  the  chart  the  evaluator  can  locate  positions  where  program  informa- 


tion is  collected  and  distributed,  (R-roles) ;  audiences  for  program  informa- 


tion, (A-roles);  program  inf Ijencers,  (C  and  A  roles);  and,  lines  of  communica- 


tion, (I-roles) 
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2.3   Describing  the  Decision  Structure  (continued) 

INFORMATION  FLOW 
A  survey  of  the  collection  and  use  of  written  information  in  program  decision- 
making is  the  third  step  in  describing  the  decision  structure.   This  description 
includes: 

charts  of  the  flow  of  forms  and  reports, 
a  review  of  items  of  data  collected, 
schedules  of  data  collection  and  reporting, 
descriptions  of  data  storage  and  retrieval, 
lists  of  audiences  for  information, 

copies  of  form  7,  Report  or  Form  Description,  for  all  program  forms  and 
reports. 

Data  from  this  survey  serves  a  dual  purpose.   It  forestalls  duplication  of 

effort  in  data  gathering.   In  combination  with  data  on  the  change  situation  and  role 

function  description,  it  helps  define  decision-making  information  needs. 

Writing  a  decision  structure  description  makes  it  possible  for  you  and  the 

evaluator  to: 

examine  the  relationship  of  the  organizational  and  decision  structures, 

locate  major  change  influences  affecting  the  program. 
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2.3  Describing  the  Decision  Structure  (continued) 

obtain  information  for  writing  job  descriptions, 

obtain  a  calendar  of  major  reporting  dates, 

locate  program  audiences,  and 

determine  types  of  information  to  be  supplied  to  them. 
A  study  of  a  decision  structure  could  go  into  much  greater  detail. 
The  list  described,  however,  includes  items  which  are  considered  basic  to 
implementation  of  ISDM. 
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2.3  Describing  the  Decision  Structure  (continued) 


WORK  BREAKDOWN  FOR  DETERMINING  THE  DECISION  STRUCTURE 


Who 


Does  What 


How 


Evaluator 


1.  Contacts  program  personnel 
to  arrange  delivery  of  forms 
7  and  8,  delivers  forms,  and 
collects  completed  forms 

2.  Records  staff  positions 

3.  Organizes  staff  positions 
on  a  chart 


By  telephone  or 
in  person 


Using  form  5 

Using  classification 

system: 

R — Responsibility  for 

initiating 
I--Informs  others 
A — Approves 
C--Consults 


4.   Adds  objectives  and  purpose 
statements  (see  Section  2.4) 


On  form  8 


Combines  influencers  and 
ratifiers  from  form  8  with 
report  of  program  function 
or  change  situation,  form  9 

Files  forms 


By  compiling  a 
composite  list 


According  to  plan 
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2.3  Describing  the  Decision  Structure  (continued) 


WORK  BREAKDOWN  FOR  DETERMINING  THE  CHANGE  SITUATION 


Who 


Does  What 


How 


Evaluator 


Obtains  form  designed  to  report 
program  function  or  change 
situation  and  obtains  program 
boundaries  for  the  program 
under  study 

Records  program  function 


According  to 
filing  policy 


According  to  the 
requirements  of 
form  9 


3.   Records  statement  of  purpose 
and  funding  source 


By  attaching  copy 
of  list  of  purposes 
and  recording  funding 
source  on  form  9 


4.  Identifies  questions 
important  to  evaluation 
planning 

5.  Obtains  list  of  objectives 
(see  Section  2.4) 


By  reviewing 
change  situation 


According  to  filing 
policy 


Records  objectives 


By  attaching  copy  of 
list  to  form 


7.   Analyzes  purpose,  statement, 
and  objectives 


By  reading  to  deter- 
mine modes  of  influence, 
units  and  objects  of 
change,  and  referring 
to  discussion  of  change 
situation 


Records  influences  for  change, 
units  and  objects  of  change 
for  each  statement  of  purpose 
and  objective 

Files  form 


According  to  the 
requirements  of  form  9 


According  to  plan 
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(Form  completed  as  REPORT  OR  FORM  DESCRIPTION 

an  illustration.)  FORM  7 

INSTRUCTIONS:   Complete  this  form  for  each  form  or  report  that  originates  or 
terminates  with  you,  or  passes  through  your  hands.   Please  obtain  a  copy  of  that 
form  or  report  and  submit  attached  to  this  instrument. 

RETURN  TO  EVALUATOR  BY:     Sp.ptP.mbp.r   l7> ,    1977> 

(date) 

Complete  title  of  form  or  report:    QU'XPterlii  Finanoial  Reyort 


Form  or  report  number  (if  applicable) 1.972-S 


Where  does  report  originate?     School  Distriot  Finanoial  Offioev 


Where  is  the  final  destination?    Beard  of  Trustees 


(indicate  all  recipients  of  the  report) 

Is  there  any  part  of  this  form  or  report  required  by  law?   Yes  \  \   No  \  x\ 

If  yes,  indicate  items  on  the  report  and  specify  law 


Frequency  of  submission Four  tines  a  ijear 


Explain  purpose  of  report  or  form  and  show  route:  (use  back  if  necessary)_ 

Aaacunts  fcr  income  and  expenditures  cf  district  funds, 

Rcvtinci:     Financial  Officer 

Superi  ntenden  t 


Bocwd  of  Trustees 


Date: 7/2d^7i      Completed  by: Marvin  E.    Accountant 
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2.4   Constructing  Objectives 


A  reservoir  is  constructed  for  known  purposes;   program  purposes 


become  known  through  the  formulation  of  clear  objectives. 


ACTIVITY:   The  evaluator  will  assist  in  identifying  or  constructing 


objectives  for  each  level  of  the  program. 


CONSIDERATIONS:   For  purposes  of  evaluation,  objectives  should  be 


stated  in  the  following  ISDM  format: 


1.   People  variable 


The  person  who  is  expected  to 


change . 


2,   Action  variable 


The  behavior  he  is  to  perform. 


3.   Program  variable_ 


The  setting  or  training  which 


will  prepare  him. 


4.   Measurement  technique_ 


_The  method  or  technique  by  which 


actual  performance  will  be  measured. 


5.   Criteria  of  success 


_The  expected  outcome;  an  accept- 


able level  of  performance. 
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2. A   Constructing  Objectives  (continued) 


An  objective  is  a  statement  of  program  purpose  or  intent.  Because 


sound  objectives  are  crucial  to  the  evaluation  process,  the  evaluator 


will  work  very  closely  with  your  organization  at  this  point. 


Decision-making  personnel  at  each  level  of  a  program  will  be  asked 


to  write  objectives,  using  ISDM  sample  form  10.   The  evaluator  will  review 


these  written  objectives  to  determine  if  they  meet  format  requirements, 


If  any  of  the  format  elements  are  missing,  he  will  make  revisions.  He 


will  then  confer  with  your  organization  to  determine  if  the  rewritten 


objectives  are  accurate, 


The  following  is  a  sample  objective: 


A 

nine-year-old 

student 

in  the 

tutorial 

reading 

program 

wl 

11 

demonstrate 

his 

abi 

lity  to 

read  by 

reading 

a  100 

word 

passage 

ora 

Uy 

with 

no  more  than  fi 

Lve  mistakes .      | 

1.   People  variable:   a  nine-year-old  student 


2.   Action  variable:   will  demonstrate  his  ability  to  read 


3.   Program  variable:   tutorial  reading  program 


4.   Measurement  techniques:   reading  a  100  word  passage  orally 


5.   Criteria  of  success:   no  more  than  five  mistakes 
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2.4  Constructing  Objectives  (continued) 

MEASUREMENT  ECHNIQUE 

Because  the  application  of  proper  measurement  technique  to  a 
corresponding  objective  is  a  specialized  skill,  this  part  of  the  objective 
is  not  always  formulated  by  the  decision-maker.   If  the  decision-maker  does 
not  specify  a  technique,  it  will  be  determined  by  the  evaluator. 

Measurement  technique  may  be  applied  to  three  domains  of  human 
learning  behavior  illustrated  on  the  cube  at  the  end  of  this  section: 
cognitive,  affective  and  psychomotor.   The  cognitive  domain  represents 
recall,  recognition  or  expression  of  intellectual  skills  and  abilities. 
The  affective  domain  represents  interests,  attitudes  and  values.   Manual 
and  motor  skills  are  included  in  the  psychomotor  domain.   These  three 
domains  are  referred  to  as  action  variables. 

Methods  or  techniques  for  measuring  action  variables  are  the  res- 
ponsibility of  the  evaluator.   For  example,  he  might  suggest  an  achievement 
test  to  measure  performance  in  the  congitive  area,  attitude  scales  for 
measurement  in  the  affective  area,  and  in  the  psychomotor  area,  the  evaluator 
might  suggest  observation  systems,  rating  scales,  or  checklists,   ^feasure- 
ment  technique  will  identify  actual  behavior  demonstrated  for  a  given 

objective. 
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2.4   Constructing  Objectives  (continued) 


CUBE  CLASSIFICATION: 
SEEING  FORMAT  ELEMENTS  IN  THREE  DIMENSIONS 


It  is  important  that  program  personnel  understand  the  focus  of  their  objec- 
tives.  One  way  to  focus  them  is  to  classify  the  objective's  format  elements  on 
a  three-dimensional  cube.   The  three  dimensions  represent  people,  action  and 
program  variables.   The  cube  also  provides  a  means  of  illustrating  actual  or 
expected  outcomes  within  the  action  variable.   The  results  of  the  measurement 
technique  are  actual  outcomes,  and  criteria  of  success  are  the  expected  outcomes. 
In  the  sample  reading  objective,  the  expected  outcome  is  that  the  student  will 
read  100  words  orally  with  no  mote  than  five  mistakes.   The  actual  outcome  may 
be  that  the  student  will  make  either  more  or  less  than  five  errors  in  reading 
the  100  words. 

The  cube  may  help  to  visualize  the  interrelationships  among  the  basic  parts 
of  an  objective.  The  format  elements  are  listed  on  the  following  pages  together 
with  illustrations  of  their  positions  on  the  cube. 

If  the  expected  behavior  differs  from  the  actual,  measured  results,  this 

information  may  be  useful  in  determining  future  objectives  and  program  direction. 

Or,  if  in  classification,  the  objectives  do  not  fall  within  the  area  your  program 

is  seeking  to  improve,  new  objectives  will  have  to  be  constructed. 
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CUBE  CLASSIFICATION: 
SEEING  FORMAT  ELEMENTS  IN  THREE  DIMENSIONS 


PEOPLE  VARIABLES 
The  shaded  area  in  the  face  of  this  cube  shows: 
1.   People  (student)  "...nine-year-old  student " 
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CUBE    CLASSIFICATION: 
SEEING  FORMAT  ELEMENTS   IN  TliREE   DIMENSIONS 


.^^ 


-^-'^^ ''.^^/y^?7. 


n^ye/ 


•«<b 


ACTION  VARIAELES 
The  shaded  area  on  the  top  of  the  cube  represents: 
2,   Action  (cognitive)  "...will  demonstrate  his  ability  to  read..." 
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CUBE  CLASSIFICATION: 
SEEING  FORMAT  ELEMENTS  IN  THREE  DIMENSIONS 


1/  I  i  A  cr/ 


PROGRAM  VARIABLES 
The  shaded  area  on  the  side  of  the  cube  represents: 
3,  (Program  activity)  "...the  tutorial  reading  program..." 


59 


CUBE   CIASSIFICATION 
SEEIN'C   FORMAT   ELEMENTS    IN   THREE    DIMENSIONS 


THREE  VARIABLES 


Pulling  the  identified  smaller  cube  from  the  larger  one  illustrates 
the  three  areas  covered  by  the  objective: 

1.  People   -   Students 

2.  Action   -    Cognitive 

3.  Program   -  Activities 
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CUBE  CLASSIFICATION: 
SEEING  FORMAT  ELEMENTS  IN  THREE  DIMENSIONS 


PSYaiGMOTGR 


AFFECTIVE 


;nitive 


ORGANIZATION 


activities 
or  content 


METHODS 


FACILITIES 


COST 


The  remaining  format  elements  of  an  objective  are  shaded  above: 

easurement  technique  (actual  outcome),  "... [results  of |  a  100  wo 
oral  reading  tesTI  .  .  .  " 


4.   Me 


rd 


Lng  test] 
5.   Criteria  of  success  (expected  outcome),  "...less  than  five  mistakes, 
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2,4  Constructing  Objectives  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator     1.   Requests  written  objectives 

from  personnel 

2.  Reviews,  rewrites  objectives 
and  suggests  measurement 
techniques 

3.  Confers  with  appropriate 
personnel  about  rewritten 
objectives 

4.  Classifies  and  records 
objectives  by  format  elements 

5.  Files  completed  objectives 


Using  form  10 


Using  ISDM  format 
for  writing 
objectives 

By  scheduling 
appointment 


Using  the  cube 


According  to  program, 
project  or  management 
level 
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PROGRAM  OBJECTIVE  REQUEST  —  FORM  10 
(This  is  a  sample  form) 


1.  Program  Title 

"AIM"     (AahieJement  Improvement  in  Mathematics) 

2.  Objectives 

Please  attach  a  list  of  objectives  fitting  the  following  format  as 
closely  as  possible. 

An  objective  is  a  statement  of  purpose  or  intent  that  includes: 
1)  people  variable,  2)  action  variable,  3)  program  variable, 
4)  measurement  technique  and  5)  criteria  of  success. 

a)  "People  variable"  identifies  the  per8on(s)  who  will  be 
the  subject(8)  of  the  objective  statement  (WHO?). 

b)  "Action  variable"  identifies  what  the  subject  variable 
will  be  expected  to  do  (WILL  DO  WHAT?). 

c)  "Program  variable"  identifies  the  dimension  in  which  the 
subject  variable  will  perform  (WHERE?). 

d)  "Measurement  technique"  identifies  the  method  and/or 
instrument  that  will  be  used  to  determine  whether  the 
performance  as  described  was  accomplished.   (Because  of 
the  difficulty  in  applying  the  proper  measurement  technique 
with  a  corresponding  objective,  the  measurement  method  does 
not  have  to  be  developed  by  the  objective  writer.) 

e)  ''Criteria  of  success"  predetermines  a  level  of  accomplish- 
ment which  will  be  deemed  acceptable. 

"A  nine-year-old  student  in  the  tutorial  reading 
program  will  demonstrate  his  ability  to  read  by 
reading  a  100  word  passage  orally  with  no  more  than 
five  mie, takes.  " 
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2.5   Constructing  Questions 

To  assure  water  quantity  and  quality,  a  reservoir  must  be  checked. 
Is  the  reservoir  serving  those  purposes  for  which  it  was  designed? 
Questions  about  education  programs  are  formulated  and  answered  to  check 
whether  objectives  are  being  met  and,  if  not,  to  determine  possible 
reasons  for  failure. 

ACTIVITY:   In  cooperation  with  the  decision-maker,  the  evaluator 
will  construct  questions  to  be  answered  about  program  objectives. 

CONSIDERATIONS:   Question  construction  is  a  creative  effort 
accomplished  by  Interaction  between  the  evaluator  and  the  decision- 
making staff.   Questions  are  devised  to  analyze  the  effectiveness  of  your 
program  objectives. 

Information  collected  in  previous  sections  (e.g..  Sections  2,1 
through  2.4)  can  be  used  to  help  construct  these  questions.   The  ISJM 
cube  can  also  aid  in  the  formulation  of  relevant  questions.   For  example, 
the  obvious  question  for  the  reading  objective  is  "Did  the  student  read 
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2.5   Constructing  Questions  (continued) 


100  words  orally  with  no  more  than  five  mistakes?" 


SAMPLE  QUESTIONS:   THE  READING  OBJECTIVE 


A  nine-year-old  (third  grade)  student 
will  demonstrate  his  ability  to  read, 
by  reading  100  words  orally  with  no  more 
than  five  mistakes. 


If  the  objective  was  accomplished,  the  decision-maker  might  want 


to  know  how  the  program  variables  affected  the  achievement  of  the  objective. 


1.  Organization:   Did  the  organizational  pattern  for  teachers  and  students 

help  to  achieve  the  objective? 

2.  Activities  or  content:   Did  the  teacher  demonstrate  cognitive  behavior 

in  reading  content  that  will  insure  achievement  of  the  objectives? 


3.  Methods:   What  instructional  methods  were  used  to  develop  students' 
oral  reading  skills? 


Facilities:   Were  reading  laboratory  facilities  adequate  for  the 
achievement  of  the  objective? 


5.   Cost:   How  much  did  it  cost  to  achieve  this  objective? 
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2.5  Constructing  Questions  (continued) 

Question  construction  Is  limited  only  by  available  Information  and 
the  creative  ability  of  the  evaluator  and  decision-maker.  Available 
Information  depends  on  Interaction  between  staffs,  while  creative 
ability  will  be  tested  when  you  and  the  evaluator  review,  revise  and  rewrite 
the  questions  formulated  from  program  objectives. 

Special  care  should  be  used  in  phrasing  the  final  questions  since 
the  evaluator  will  develop  alternative  answers,  alternative  actions  and 
criterion  variables  from  them. 

Working  on  ISDM  form  12,  the  evaluation  staff  will  consider  and 
record  as  many  alternative  answers  as  possible.  Alternative  actions. 
Intended  for  the  evaluator,  are  developed  in  response  to  the  alternative 
answers.   Criterion  variables  are  measurement  techniques  which  can  be  used 
to  answer  questions.   These  are  often  developed  In  consultation  with 
technical  personnel. 
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2.5  Constructing  Questions  (continued) 


WORK  BREAKDCWN 


Who 


Does  What 


How 


Evaluator 


Obtains  and  records  program 
title,  program  definition 


By  reviewing  Section  2.1 
and  entering  on  form  10 


2.  Obtains  and  records  action 
verb,  objectives  and  their 
sourceCs) 


By  reviewing  Section  2.4 
and  entering  on  form  10 


3.   Schedules  visits  with 
decision^nakers  and 
meets  with  then 


By  making  appointments 
and  discussing  forces 
that  may  affect  program 


4.   Constructs  and  records 
questions 


Using  information  from 
meetings  with  decision- 
makers and  recording  on 
form  10 


Schedules  meetings  and 
meets  with  decision- 
makers 


By  making  appointments 
and  discussing  questions 
that  "emerged"  and  were 
constructed 


Revises  questions 


By  considering  infor- 
mation from  second 
meeting  with  decision- 
makers 


7.   Develops  alternative 
responses 


By  preparing  list  of 
possible  alternatives 
and  recording  on 
form  11 
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2.5  Constructing  Questions  (continued) 


Who 


Does  What 


How 


Evaluator 


Develops  and  records 
alternative  activity 
for  evaluation 


By  referring  to 
alternative  answers 
developed  in  Section 
2 . 1  and  recording  on 
form  11 


9.   Suggests  and  records 
measurement  for 
criterion  variable 


By  consulting  with 
instrument  specialist 
and  recording  on 
form  11 


10.   Files  completed  form 


With  file  to  be  used 
in  instrumentation 
development.  Section 
3.2 
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PHASE  III  -  GATHERING  THE  DATA 
3.1  Identifying  Data  Sources 

Samples  can  reveal  the  quality  of  a  reservoir's  water,  while  sampling 
a  population  can  reveal  characteristics  helpful  in  answering  the  questions 
constructed  for  program  objectives. 

ACTIVITY:  The  evaluator  will  plan  procedures  for  gathering  data  to 
be  used  in  answering  questions  about  program  objectives. 

CONSIDERATIONS:  He  will  identify  what  data  is  needed,  the  data  source, 
data  form,  population  involved  and  sample  size  required.  He  may  also  need 
to  Identify  a  sampling  procedure. 

If  the  data  needed  concerns  people  or  programs,  the  evaluator  will 
determine 

data — ^What  data  is  to  be  obtained? 

data  source—Where  is  the  data?  Who  has  it?  Who  is  contacted  in  order 

to  obtain  it?  If  it  is  not  available,  a  potential  source 

must  be  located. 

data  form— Is  the  data  recorded  or  non-recorded?  Is  it  on  file,  compu-er 
cards,  etc.? 
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3.1  Identifying  Data  Sources  (continued) 

In  addition,  If  the  needed  data  concerns  people,  he  will  determine 
population — ^Who  are  the  Individuals  or  elements  Involved?  What  Is  the  total 

number  of  Individuals  Involved? 
sample  size  requlred--What  Is  the  number  of  people  within  a  population 

needed  to  gather  requested  Information? 
sampling  procedure—What  Is  the  suggested  sampling  method? 

An  Important  consideration  In  gathering  data  about  large  groups  of 
people  Is  that  Information  concerning  characteristics  of  all  Individuals  In 
the  group -may  be  unavailable  or  too  expensive  to  collect.   The  fundamental 
assumption  underlying  use  of  statistics  Is  that  reliable  statements  about 
populations  can  be  made  by  examining  characteristics  of  a  smaller  number  of 
Individuals  who  are  representative  of  the  population  as  a  whole. 

Three  principles  are  fundamental  to  proper  utilization  of  statistics: 
(1)  the  smaller  number  of  individuals,  called  the  sample,  must  be  chosen  In 
a  manner  which  assures  that  they  are  Indeed  representative  of  the  larger  group 
called  the  population; 

72 


3.1  Identifying  Data  Sources  (continued) 

(2)  the  sample  must  include  a  large  enough  number  of  these  representative 
individuals ;  and 

(3)  the  proper  statistical  procedure  must  be  chosen  to  correspond  to  the 
kind  of  data  which  is  being  gathered. 

SAMPLING 

In  most  situations  the  evaluator  will  face,  he  will  have  a  choice  of 
methods  in  selecting  a  sample  which  is  representative  of  the  population.   If 
he  can  select  a  sample  which  is  truly  drawn  at  random,  and  if  the  sample  is 
large  enough,  he  can  be  confident  of  statements  made  about  the  population. 
In  a  randomly  selected  sample,  each  individual  theoretically  has  an  equal 
chance  to  be  included  in  the  sample.   In  an  instance  wh^re  the  population  is 
finite  (for  example,  all  eighth  grade  typing  students  in  a  school),  a  simple 
and  reliable  method  of  insuring  randomness  is  to  employ  a  table  of  random 
numbers. 

In  some  situations,  the  evaluator  may  be  confronted  with  a  population 
so  large  as  to  render  simple  random  sampling  unwieldy.   Suppose  that  he  intends 
to  discover  how  voters  in  a  community  regard  a  proposed  school  bond  levy.  Here, 
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3.1  Identifying  Data  Sources  (continued) 

the  appropriate  technique  may  be  stratified  random  sampling,  in  which  the  eval- 

uator  must  pick  his  sample  so  that  the  individuals  included  in  it  are  in  the 

same  proportions  as  they  occur  in  the  population.   For  example,  he  may  have 

to  know  what  percentage  of  the  population  is  female,  over  50,  earns  $10,000 

or  more  annually,  is  Protestant,  or  has  any  other  characteristic  relevant 

to  his  study. 

Often,  however,  the  evaluator  may  not  be  able  to  pick  a  sample,  but 
rather  must  work  with  one  which  is  available  to  him.  In  this  case,  he  must 
attempt  to  correct  for  existence  of  any  overrepresentation  of  categories  which 
he  can  identify  in  the  sample. 

ERRORS 

Even  vihen  the  evaluator  has  chosen  a  sample  at  random,  he  cannot  make 
reliable  statements  about  the  population  unless  the  sample  size  is  large 
enough.   For  example,  one  person  may  be  chosen  purely  at  random,  but  his 
characteristics  may  tell  very  little  about  a  group  of  100  bird  watchers  to 
which  he  belongs.   Consequences  of  using  a  non-representative  sample  were 
illustrated  by  a  1948  presidential  election  prediction.  After  voters  who 
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3.1  Identifying  Data  Sources  (continued) 

had  telephones  were  polled,  Dewey  was  picked  to  beat  Truman.  Although  the 
prediction  was  correct  for  voters  who  had  telephones,  it  was  incorrect  £or 
the  total  population.   Failure  to  pick  a  truly  representative  sample  of  the 
voting  population  invalidated  the  prediction. 

In  general,  the  smaller  the  population,  the  greater  the  proportion  which 
must  be  included  in  the  sample.  Whenever  the  sample  is  too  small,  the  effect 
of  sampling  error  may  make  inferences  unacceptable.   This  error  is  the 
difference  between  the  population's  real  characteristics  and  those  attributed 
to  it  from  analyzing  the  sample. 
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3.1  Identifying  Data  Sources  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


1.  Reviews  sections  on 
boundaries  (2,1), 
people  and  program 
variables  (2.2), 
decision  structure 
(2.3),  objectives 
(2.4),  questions  (2.5) 

2.  Records  information  that 
exists  or  is  needed  for  each 
Section  reviewed,  records 
source  and  form  of  information, 
records  population 


By  referring  to  file 
containing  information 
on  each  Section 


Using  form  13 


3.   Determines  and  records 
sample  size  required  and 
sampling  procedure 


Using  form  13 
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(This  is  a  sample  form) 


DATA  SOURCE  FORM 
FORM  13 


DATA 

(Instrument  results, 

unanswered  questions,  etc.) 


Rtud^nt'e  individual  I.Q. 


asorss 


SOURCE 

(Where  Is  data?  Who  has 

It?   Contact  for  obtaining?) 


Student  file.  Mr.   Jones, 

Guidance 


FORM  OF  DATA 
(Is  it  recorded  or 
non-recorded,  on  file, 
on  computer  cards,  etc.?) 


9th  grcde  math  students  in 
Lyndale   ...Computer  printout 

.mt \ 


SAMPLE  SIZE  REQUIRED 
(Number  needed  to  gather 
requested  information.) 


.5iL 


SAMPLING   PROCEDURE 
(Short  description  of 
suggested  method.) 


Sa%eat  ever\j  lOth  student 
from  an  alphdbetio  list 
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3.2  Selecting  Instruments  for  Evaluation 

If  a  check  on  the  water  quality  of  a  reservoir  were  needed,  an 
appropriate  instrument  would  be  applied  in  making  the  test.   If  it  were 
necessary  to  isolate  a  certain  type  of  bacteria  present  in  the  water, 
another  kind  of  instrument  would  be  utilized.  Instruments  vary  with  the 
kind  of  data  being  gathered. 

ACTIVITY:  The  evaluator  will  select  or  design  valid  and  reliable 
instruments  for  use  in  gathering  needed  program  data. 

CONSIDERATIONS:  Specifications  for  evaluation  instruments  (e.g., 
questionnaires,  aptitude  tests,  achievement  tests,  statements  of  opinions) 
will  include 
1.   statement  of  needed  data, 


2.   statement  of  instrument's  reliability  (does  it  gather  similar  data 
consistently?)  and  validity  (does  it  gather  pertinent  data?) ,  and 


3.   statement  regarding  pilot  test  results. 

If  specifications  cannot  be  met  by  existing  instrimients,  construction 
of  new  instruments  becomes  necessary.   New  evaluation  instruments  should  be 
pilot  tested  before  field  use.   Periodic  review  of  instruments  and  their 
ability  to  meet  the  criteria  of  validity  and  reliability  is  also  recommended. 
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3.2  Selecting  Instruments  for  Evaluation  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


Determines  needs  for 
Instruments 


By  reviewing  Sections 
1.2  and  2.1  through 
2.5 


2.  Obtains  and  reviews 
existing  instruments 

or 

3.  Constructs  needed 
instruments  in  cooper- 
ation with  instrument 
specialist 

A.  Pilot  tests  instruments, 
if  necessary,  on  similar 
populations 

5.  Files  completed  instrument 


By  checking 

a.  data  needs 

b.  reliability  and 
validity 

c.  pilot  results 
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3.3  Gathering  Conditions 

Data  is  used  in  different  ways,  Just  as  a  reservoir  is  used  for 
water  and  power  supply,  food  control,  irrigation  and  recreation.  A 
schedule  will  be  devised  at  this  time  for  gathering  data  to  answer 
various  program  needs. 

ACTIVITY:   The  evaluator  will  develop  a  schedule  for  gathering  the 
data  to  be  used  in  answering  questions  about  program  objectives. 

CONSIDERATIONS:   Data  to  be  gathered  has  been  Identified;  the 
schedule  for  data  gathering  will  be  determined  at  this  point.   The  follow- 
ing items  might  be  included  in  the  schedule,  recorded  atid  filed  (see  sample 
form  14  on  page  81) : 

name  of  program, 

name  of  instrument, 

who  will  gather  the  data, 

sampling  unit  required, 

time  required  for  administration, 

approximate  date  of  administration,  and 

setting  for  instrument  administration. 
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3.3  Gathering  Conditions  (continued) 

In  developing  a  schedule,  the  evaluator  will 

(1)  review  instructions  for  gathering  data, 

(2)  determine  the  number  of  persons  necessary  to  gather  the  data, 

(3)  locate  a  suitable  place  for  Instrument  administration,  and 

(4)  consider  program  policies  when  deciding  who  will  be  responsible 
for  administering  an  instrument. 
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3.3  Gathering  Conditions  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


4. 
5. 


Obtains  all  instruments 
which  have  been  constructed 
in  Section  3.2 

Determines  and  records 
who  should  administer 
what  instrument 

Reviews  sampling  unit 
procedure  from  Section  3.1 
and  records  sampling  unit 
required 

Records  time  required  for 
administration 

Determines  and  records 
setting  for  administration 
of  instruments 

Constructs  schedule  for 
administration  of 
instruments 


By  retrieving  them  from 
the  "Instrument  File" 


Using  form  14 


Using  form  14 

Using  form  14 
Using  form  14 


By  referring^  to 
information  notes  from 
the  above  steps 
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3.4  Organizing  Data 

If  a  reservoir  has  either  too  much  or  too  little  water,  Its  function 
Is  Impaired.  The  organization  of  data  to  be  gathered  must  be  controlled 
if  data  Is  to  be  functional. 

ACTIVITY:   The  evaluator  will  specify  the  amount  and  grouping  of 
needed  program  data  and  will  devise  logical  means  by  which  to  record 
that  data. 

OONSIDEBATIONS :   To  control  the  amount  of  data  gathered,  to  determine 
what  grouping  will  best  meet  the  Intended  use  of  the  data  and  to  devise 
logical  systems  for  filing  and  retrieving  that  data,  three  procedures  must 
be  completed: 

1.  stating  the  unit  of  organization, 

2.  establishing  storage  and  retrieval  requirements,  and 

3.  establishing  quality  control  procedures. 
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3.4  Organizing  Data  (continued) 

UNIT  OF  ORGANIZATION 

The  detail  and  coding  format  of  data  must  be  selected  with  you,  the 

decision-maker,  in  mind.   The  smallest  data  unit  required  is  related  to 

your  information  needs  and  intended  use;  it  is  the  basis  for  storage, 

retrieval,  analysis  and  reporting.   For  example,  if  an  analysis  of  each  item 

on  a  test  is  desired,  this  analysis  could  be  the  smallest  data  unit  to  be 

stored.  A  frequency  distribution  might  be  sufficient  record  of  performance  by 

grade  level  or  program. 

STORAGE  AND  RETRIEVAL 

Storage  and  retrieval  requirements  include 
coding  format  for  storage, 

storage  procedures,  and 

storage  and  retrieval  facilities. 
The  evaluator  may  work  with  data  processing  representatives  to  develop 
a  data  storage  plan.   If  a  manual  filing  system  is  used,  each  stored  form, 
chart  or  listing  should  be  fully  labeled;  a  master  cross-reference  index  to 
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3.4  Organizing  Data  (continued) 

data  holdings  will  be  necessary.  In  all  systems,  whether  manual  filing 
or  computer  storage,  it  must  be  remembered  that  personnel  who  retrieve  the 
data  will  not  necessarily  be  those  who  stored  it  originally. 

The  evaluator  should  also  specify  for  data  processors  the  storage 
and  retrieval  facilities  to  which  he  has  access.   Specifying  these  facilities 
allows  the  evaluator  and  data  processors  to  plan  their  activities  and  to 
identify  supportive  services  which  might  be  needed,  such  as  a  faster  printer 
to  supply  immediate  feedback. 

QUALITY  CONTROL 

To  build  quality  control  into  the  data  organization  system,  the 
evaluator  and  data  processors  need  to  establish  editing  procedures  and 
reliability  checks.   Quality  control  and  editing  are  particularly  important 
when  critical  decisions  -  e.g.,  admission  of  students  to  college  -  will  be 
based  on  the  data  -  e.g.,  students'  high  school  grades. 
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3.4  Organizing  Data  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


Evaluator 
and  Data 
Processing 
Representative 

Data  Processing 
Representative 


Establishes  how  small 
the  data  unit  must  be  to 
satisfy  audience  needs 


Set  coding  format 


3.  Establishes  coding 


By  referring  to  Sections 
2.1  through  3.3  to  determine 
information  needs  and  infor- 
mation recipients 

By  working  together  to  meet 
the  decision'-maker's  needs 


By  allowing  for  growth, 
rewrites,  redesigns  and 
hardware  and  software 
restrictions 


Evaluator  and 
Data  Processing 
Representative 


Evaluator  and 
Data  Processing 
Representative 


4.  Document  storage 
procedure 


5.  Specify  storage 
facllltiea 

6.  Specify  retrieval 
procedures  and 
facilities 


7.   Build  quality  control 


By  classifying  information 
according  to  some  subject 
content  and  indexing 
Information  types 

By  determining  volume  of 
storage  space  needed 

By  drawing  tentative  output 
formats,  getting  approval 
for  output  layouts  or  system 
to  draw  tentative  output 
formats,  providing  input  lay- 
outs and  new  system  flow,  and 
getting  approval  for  input 
and  system  flow 

By  establishing  editing 
procedures  and  reliability 
checks 
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PHASE  IV  -  PROVIDING  INFORMATION 
4.1  Analyzing  Data 

A  reservoir  supplies  energy  potential;  water  is  used  to  turn  the  turbine 
which  produces  power.   Data  flowing  into  the  ISDM  reservoir  is  transformed 
into  decision-making  information  through  data  analysis. 

ACTIVITY:   To  satisfy  the  information  needs,  the  evaluator  will  deter- 
mine and  apply  methods  of  data  analysis. 

CONSIDERATIONS:  Data  analysis  refers  to  aggregation  or  groupings  of 
data  for  decision-making.  Aggregation  methods  for  the  two  basic  types  of 
data  include: 


1.   descriptive  or  comparative  data--aggregation  into  group  or  summary 

data  such  as  frequency  distributions  and  statistics  of  central  tendency 
(average,  middle  score,  most  frequent  score  and  variation),  and 


2.   explanatory  or  predictive  data — aggregation  into  performance  level  on 
objectives  or  other  variables  being  measured. 


The  evaluator  should  be  careful  to  avoid  losing  valuable  information 
while  forming  aggregates.  An  example  of  over-aggregation  might  be  reduction 
of  all  classroom  performance  scores  to  one  mean  (arithmetic  average)  for  the 
group;  then  if  individual  pupil  performance  data  were  later  desired,  only 
classroom  data  summaries  would  be  available. 
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4.1  Analyzing  Data  (continued) 

The  purpose  of  analysis  Is  to  transfer  raw  data  into  information.   Two 
criteria  must  be  met  for  this  process : 

1.  original  data  must  contain  desired  information  or  have  a  known  relation- 
ship to  it,  and 

2.  desired  information  must  be  presented  In  a  form  understandable  to  the 
audience.  For  example,  it  might  be  necessary  to  prepare  graphs  or 
charts  rather  than  using  mathematics  or  technical  language. 

Four  categories  of  data  analysis  have  been  identified  and  some  common 
needs  linked  to  them.   Form  15  will  be  used  to  record  descriptions  of 
analysis  methods. 

DESCRIPTIVE  ANALYSIS 

Descriptive  analysis  is  a  depiction  of  what  exists,  such  as  the  number 
of  pupils  in  a  school,  their  ages,  sex,  present  level  of  performance  or  non- 
performance. This  type  of  analysis  is  not  intended  to  examine  relationships 
or  to  make  comparisons. 

Descriptive  data  is  conveyed  in  either  numerical  or  expository  (written 
narrative)  form,  depending  on  the  situation.   The  numerical  form  is  used 
where  data  is  already  in  the  form  of  numbers  or  is  easily  converted  to  numbers. 
Analysis  of  numerical  descriptive  data  may  be  limited  to  simple  counts  or 
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4.1  Analyzing  Data  (continued) 

tabulations  or  to  measurement  of  central  tendency.  This  type  of  analysis 

will  most  frequently  use  descriptive  statistics.  The  evaluator  or  other 

specialized  personnel  will  help  you,  the  decision-maker,  choose  a  proper 

analysis  technique. 

When  a  significant  information  loss  might  result  from  changing  data 

into  statistics,  the  expository  descriptive  form  should  be  used.   This  type 

of  analysis  is  subjective  and  involves  the  evaluator 's  value  judgments.   This 

fact  is  an  important  consideration  and  should  be  specified  when  reporting  an 

expository  analysis. 

COMPARATIVE  ANALYSIS 

Where  it  is  necessary  to  compare  two  or  more  situations,  methods  or 
processes,  comparative  analysis  is  used.   Relative  or  absolute  standards  may 
be  applied  in  this  type  of  analysis. 

Relative  standards  are  used  to  identify  differences  in  behavior.   So  that 
dependent  variables  may  be  compared,  subject  groups  must  have  at  least  one 
common,  independent  variable  (e.g.,  age,  sex,  or  socioeconomic  status).   Basically 
descriptions  of  two  or  more  groups  are  being  compared  with  one  another. 
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4.1  Analyzing  Data  (continued) 

When  absolute  standards  are  used,  one  or  more  groups  are  compared  to 
an  ideal  or  set  level  of  performance,  rather  than  compared  to  another  group. 
(For  example,  all  students  might  be  required  to  demonstrate  mastery  of  the 
multiplication  tables  at  the  end  of  the  fifth  grade.  Another  level  of 
mastery  would  be  requiring  all  students  to  score  at  least  at  the  eighth 
grade  norm  in  mathematics  prior  to  graduation.) 

Comparative  data  may  be  numerical  or  expository.  Numerical  data  should 
be  used  unless  there  is  a  possibility  that  significant  information  would  be 
lost.   Once  again,  the  subjectivity  of  expository  comparative  analysis  is 
emphasized. 

EXPLANATORY  ANALYSIS 

Explanatory  analysis  seeks  to  determine  causes  for  an  event.   For 
example,  a  comparison  of  groups  reveals  differences  in  performance.   To  what 
can  these  differences  be  attributed? 

In  explanatory  data  analysis,  the  data  collection  design  should  minimize 
the  influence  of  extraneous  variables  not  under  study.  Without  this  control. 
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4.1  Analyzing  Data  (continued) 

variables  not  under  study  interfere  with  determining  possible  cause-effect 
relationships  between  variables  which  are  being  studied. 

PREDICTIVE  ANALYSIS 

Prediction  is  concerned  with  data  that  can  reliably  predict  an  outcome, 
not  with  what  has  caused  a  certain  kind  of  behavior.   (For  example:   college 
admissions  officers  use  various  test  results  to  predict  college  performance, 
counselors  use  test  results  to  advise  students  on  potentially  rewarding 
careers,  and  early  school  admissions  are  based  on  predictions  of  readiness 
for  first  grade.) 

Analysis  of  predictive  data  usually  will  make  use  of  statistical 
techniques;  it  is  possible  that  expository  data  might  be  used  but,  again, 
the  subjective  factor  may  affect  prediction  accuracy. 
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4.1  Analyzing  Data  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Evaluator 


1.  Reviews  InforTaatlon  needed     By  referring  to 
for  decision-making  Section  2.5 


2.  Summarises  Information 
needed 


Using  form  15 


Reviews  unit  of  data 
organization 


By  referring  to 
Section  3.4 


Summarizes  unit  of 
organization 


Using  form  15 


5.   Summarizes  sampling 
unit 


Using  form  15 


6.  Selects  analysis  method 


By  using  Information 
obtained  from  the 
above  steps,  his 
experience  and 
training 


7.  Analyzes  the  data 


According  to  analysis 
method  selected 


8,  Files  form 
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SUMMARY  FOR  DATA  ANALYSIS  --  FORM  15 
(Form  completed  as  an  illustration) 

INSTRUCTIONS: 

1.  Refer  to  the  information  needed  for  decision-making. 

2.  Refer  to  the  sampling  unit  and  the  unit  of  organization  of  data. 

3.  The  information  from  1  and  2  above  will  be  reviewed  and  summarized  by 
recording  on  the  Summary  for  Information  Analysis. 


INFORMATION 
NEEDED 


SAMPLING 
UNIT 


UNIT  OF 
ORGANIZATION 


ANALYSIS 
METHOD 


le  a  student' a 
mental  ability 
8Qore  indicative 
of  mathematics 
achievement?     I) 
Student's  indivi- 
dual Henmon- 
N els  on  scores  and 
2)  SAT  Mathematics 
Achievement  scores 


Every  lOth 
student  from  an 
alphabetic  list 
of  SOO  Lyndale 
9th  grade 
students 


Individual 
"Raw  Scores" 


Analysis  of 
variance 
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4.2  Reporting  Information 

The  potential  of  the  ISEM  reservoir  la  finally  realized  %^en  reports 
flow  from  it  to  you. 

ACTIVITY:   The  evaluator  will  describe  a  plan  for  reporting  which 
Includes  assignment  of  reporting  responsibilities  within  his  team,   iden- 
tification of  report  audiences,  guidelines  for  report  preparation,  and 
recommendations  for  report  dissemination. 

CONSIDERATIONS:   Reported  information  must  communicate!   To  be 
useful,  information  should  be 


Understandable;   Does  the  information  communicate  with  the  intended 
audience? 


Reliable;  Have  procedures  for  gathering  data  been  used  equally  well  by 
different  people  to  provide  sound  information? 

Objective;  Are  biases  in  information  explicit? 

Relevant;   Is  the  Information  pertinent  to  the  decision? 

Important;   Is  the  information  significant  and  conclusive? 

Accurate;  Are  distortions  or  irregularities  in  the  Information  understood 
and  accounted  for? 

Timely:   Does  information  arrive  in  time  to  be  useful  for  decision-making? 

Pervasive:   Does  information  reach  all  decision-makers  who  have  been 
designated  as  recipients? 

Comprehensive;   Is  information  being  provided  for  all  areas  of  the  evaluation? 
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4.2  Reporting  Information  (continued) 

REPORTING  RESPONSIBILITY 

Providing  information  is  the  goal  of  ISDM.  It  is  not  necessarily, 
however,  a  terminal  activity;  it  may  be  continuous  or  periodic.   The 
evaluator— if  he  is  not  to  do  the  reporting  himself--must  be  certain  to 
assign  reporting  responsibility  to  a  person  well  acquainted  with  the  infor- 
mation system.   The  reporter  should  be  familiar  with  the  program  under  eval- 
uation and  its  personnel.   He  should  assist  in  designing  data-gathering 
techniques  so  these  will  be  compatible  with  a  coherent  report.  Since  success 
of  the  report  depends  on  the  reporter's  ability  to  assimilate  evaluation 
findings,  he  should  work  closely  with  evaluation  and  management  teams. 

AUDIENCE  IDENTIFICATION 

The  first  reporting  activity  should  be  identification  of  report 
recipients.   Report  audiences  will  be  categorized  into  two  groups,  mandated 
and  supplementary. 

Mandated  audiences  are  persons  or  organisations  requiring  a  statement 
of  the  evaluation  findings,  either  by  law  or  by  stipulation  in  the  original 
evaluation  contract.   These  may,  for  example,  be  federal  or  state  author- 
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4.2  Reporting  Infozmatlon  (continued) 

ities  or  funding  sources.  The  evaluator  should  review  the  policy  state- 
ments regarding  report  dissemination  In  order  to  identify  these  audiences 
specifically. 

Reports  may  also  be  disseminated  to  supplementary  audiences .   These 
audiences  might  include  parents,  students,  teachers  or  community  groups. 
For  a  comprehensive  reporting  program,  the  reporter  should  interact  with 
decision-makers  at  various  levels  to  identify  audiences.  He  must  make  cer- 
tain, however,  that  his  plans  to  Include  supplementary  audiences  do  not 
violate  existing  policy  for  report  distribution. 

PREPARATION  OF  REPORTS 

The  reporter  must  transmit  all  Important  findings  and  conclusions  to 
the  report  audience.  His  report  must  be  pertinent  and  useful  and  designed 
to  meet  the  needs  of  the  audience.   If  evaluation  results  are  to  be  reported 
to  more  than  one  audience,  the  possibility  of  preparing  separate  reports 
should  be  explored.   (For  example,  a  program  decision-maker  would  need  a 
detailed  report;  the  Superintendent  of  Schools  might  require  only  a  state- 
ment of  findings  and  conclusions.) 
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4.2   Reporting  Information  (continued) 

The  reporter  must  select  a  reporting  medium  which  will  effectively 
communicate  with  intended  audiences.  A  clear,  well-organized  written  report 
may  be  suitable  for  a  decision-maker;  a  concise  memorandum  may  be  an  effec- 
tive report  for  an  audience  within  the  reporter's  organization.   Television, 
radio,  or  news  publications  may  communicate  with  large  or  widespread  aud- 
iences, while  a  verbal  summary  may  be  effective  for  a  small  group.   The 
reporter  should  explore  the  value  of  these  and  other  reporting  media.   He 
must  consider  limitations  such  as  funds  budgeted  for  reporting,  degree  of 
formality  required  by  the  audience,  policy  restrictions,  and  contract  terms. 

REPORT  DISSEMINATION 

The  evaluator  will  plan  procedures  for  producing  and  transmitting 
reports.   By  referring  to  the  section  on  policy  (Section  1.2)  the  evaluator 
will  be  able  to  identify  current  reporting  policies. 

At  this  point  in  the  system,  it  is  crucial  that  reports  reach  the  right 
audiences  at  the  right  time.   In  order  to  schedule  and  stage  report  presenta- 
tions, a  production  network  must  be  planned.   Since  many  people  will  have 
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4.2  Reporting  Information  (continued) 

production  responsibilities — evaluators,  audiovisual  personnel,  data 
processors,  proofreaders,  printers,  artists,  administrative  staff,  etc. — 
careful  communication  is  essential  to  insure  successful  presentation. 

Since  reporting  and  evaluation  are  often  concurrent,  the  reporter  and 
evaluator  should  prepare  a  schedule  listing  specific  dates  when  reports  are 
to  be  available  to  their  audiences.  This  schedule  may  describe  a  series  of 
report  procedures  or  it  may  be  designed  to  provide  continual  reporting  on 
changing  situations  within  the  program.   In  any  case,  the  schedule  must 
observe  deadlines  and  time  requirements  of  report  audiences. 

Receptivity  is  an  important  part  of  reporting  and  the  channels  and 
hierarchy  of  administration  should  be  observed  to  assure  expedient  and 
acceptable  dissemination  routes.   Protocol,  as  well,  should  receive  some 
attention.  Each  situation  will  require  a  flexible  flow  of  reporting. 
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4.2  Reporting  Information  (continued) 


WORK  BREAKDOWN 


Who 


Does  What 


How 


Reporter 
from 

Evaluation 
Team 


Receives  reporting  assign- 
ments from  evaluator 

Identifies  report 
audiences  including: 

(a)  Mandated 

(b)  Supplementary 


By  meeting  with  evaluator 


By  referring  to  Sections  on 
policy  (1.2)  and  decisions 
(2.3)  and  referring  to  source 
of  funding,  authority,  etc., 
in  Sections  on  boundaries 
(2.1),  people  and  program 
variables  (2.2),  data  sources 
(3.1),  and  conferring  with 
program  supervisor  and/or 
project  director 


Depicts  levels  of 
audiences  to  predict 
comprehension  of  and/or 
preference  for  general- 
ized or  specialized 
information 


By  assisting  with  formulation 
of  organizing  data  (3.4), 
after  considering  audiences 
identified  in  Step  1 


Describes  reporting, 
including: 

(a)  Reporting  content 

(b)  Reporting  media 


By  considering  formal  vs. 
informal  delivery  and  internal 
vs.  external  consumption,  con- 
structing preliminary  table  of 
contents  and  settings  most 
conducive  for  oral,  written  or 
graphic  presentation  and  con- 
structing appropriate  formats 


Reporter 

from 

Evaluation 

Team 


Establishes  reporting 
schedule 


By  setting  deadlines  concerning 
conception,  design  and  prepara- 
tion; and  devising  guidelines 
for  production  and  delivery  of 
reports 


Prepares  reports 


By  researching,  writing, 
editing  and  designing  after 
considering  composite  criteria 
in  Steps  1  through  4 
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